Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(loki): add support for sts endpoint when using s3 buckets #13518

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

tiagoposse
Copy link
Contributor

What this PR does / why we need it:

This PR introduces the ability to override the sts endpoint used to authenticate with the AWS sdk / thanos s3 provider. It also provides a path to use STS authentication with minio operator.

As a note for those who try the minio operator STS auth: the AWS SDK adds a trailing slash when calling STS endpoint, which will cause the auth call to fail with 404. An nginx proxy can fix this the following config:

    location /sts/platform/ {
      resolver kube-dns.kube-system.svc.cluster.local;

      # Forward the request to the upstream server
      proxy_pass https://<ACTUAL ADDRESS OF THE STS SERVICE FOR MINIO OPERATOR>;
      proxy_set_header Host $host;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header X-Forwarded-Proto $scheme;

      # Preserve the original request method
      proxy_method $request_method;
    }

Which issue(s) this PR fixes:
Fixes #10751

Special notes for your reviewer:

Checklist

  • Reviewed the [CONTRIBUTING.md]
    (https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required)
  • Documentation added
  • Tests updated
  • Title matches the required conventional commits format, see here
    • Note that Promtail is considered to be feature complete, and future development for logs collection will be in Grafana Alloy. As such, feat PRs are unlikely to be accepted unless a case can be made for the feature actually being a bug fix to existing behavior.
  • Changes that require user attention or interaction to upgrade are documented in docs/sources/setup/upgrade/_index.md
  • For Helm chart changes bump the Helm chart version in production/helm/loki/Chart.yaml and update production/helm/loki/CHANGELOG.md and production/helm/loki/README.md. Example PR
  • If the change is deprecating or removing a configuration option, update the deprecated-config.yaml and deleted-config.yaml files respectively in the tools/deprecated-config-checker directory. Example PR

@tiagoposse tiagoposse requested a review from a team as a code owner July 14, 2024 18:17
@CLAassistant
Copy link

CLAassistant commented Jul 14, 2024

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ tiagoposse
❌ Tiago Posse


Tiago Posse seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
You have signed the CLA already but the status is still pending? Let us recheck it.

@github-actions github-actions bot added the type/docs Issues related to technical documentation; the Docs Squad uses this label across many repositories label Jul 14, 2024
@tiagoposse tiagoposse changed the title feat: add support for sts endpoint when using s3 buckets feat(loki): add support for sts endpoint when using s3 buckets Jul 15, 2024
@periklis
Copy link
Collaborator

periklis commented Jul 15, 2024

@tiagoposse Have you considered already incoming work like this that might fit your purpose:

In general @JoaoBraveCoding is contributing the full switch for all clients to use thanos-objstore, e.g. GCP, Azure, Swift. This is a general course we follow on among the maintainers team.

@tiagoposse
Copy link
Contributor Author

@periklis the linked PR indeed overwrites the changes I introduce here and it's definitely the correct way to go. My problem is that this PR is open for a considerable time and is quite a high impact change, so it will likely take a while to get merged.
This PR would provide this feature as a small change while his work is being reviewed.

@periklis
Copy link
Collaborator

@periklis the linked PR indeed overwrites the changes I introduce here and it's definitely the correct way to go. My problem is that this PR is open for a considerable time and is quite a high impact change, so it will likely take a while to get merged. This PR would provide this feature as a small change while his work is being reviewed.

The PR has been quite long open because we postponed it after 3.x. However we are here at good path to this merged soon, @kavirajk and @JoaoBraveCoding work in that regard closely.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
size/M type/docs Issues related to technical documentation; the Docs Squad uses this label across many repositories
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Allow S3 Storage Provider to support sts_endpoint
3 participants